CONSTRUCTING MULTIDIMENSIONAL PERIODIC CONTINUED 
FRACTIONS IN THE SENSE OF KLEIN 



O. N. KARPENKOV 

Abstract. We consider the geometric generalization of ordinary continued fraction to the 
multidimensional case introduced by F. Klein in 1895. A multidimensional periodic continued 
fraction is the union of sails with some special group acting freely on these sails. This group 
transposes the faces. In this article, we present a method of constructing "approximate" fun- 
damental domains of algebraic multidimensional continued fractions and an algorithm testing 
whether this domain is indeed fundamental or not. We give some polynomial estimates on 
number of the operations for the algorithm. In conclusion we present an example of funda- 
mental domains calculation for a two-dimensional series of two-dimensional periodic continued 
fractions. 
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Introduction, definitions, and background 



The main goal of this paper is to introduce a new method for constructing the fundamental 
domains of multidimensional periodic continued fractions in the sense of Klein. The problem of 
generalizing ordinary continued fractions to the higher-dimensional case was posed by C. Her- 
mite [9] in 1839. A large number of attempts to solve this problem leads to the birth of several 
different remarkable theories of multidimensional continued fractions. In this paper we con- 
sider the geometrical generalization of ordinary continued fractions to the multidimensional case 
represented by F. Klein in 1895 and published by him in [15] and [16]. 

A number of properties for ordinary continued fractions possesses multidimensional analo- 
gies. H. Tsuchihashi [38] found the connection between periodic multidimensional continued 
fractions and multidimensional cusp singularities. J.-O. Moussafir in [27] and O. German in [7] 
described relationship between sails of multidimensional continued fractions and Hilbert bases. 
M. L. Kontsevich and Yu. M. Suhov discussed the statistical properties of the boundary of a 
random continued fraction in [17]. The papers [35] and [36] by B. F. Skubenko and [8] by 
O. N. German are dedicated to the generalization of ordinary continued fractions with bounded 
above integer lengths of edges (the numbers that corresponds to such continued fractions are the 
numbers with the worst possible rational approximations). For the classical theory of ordinary 
continued fractions we refer to the book [10] by A. Ya. Hinchin. V. I. Arnold suggested to inves- 
tigate the geometry and combinatorics of continued fractions (it contains the study of properties 
for such notions as affine types of the faces of the sails, their quantities and frequencies, integer 
angles between the faces, integer distances, volumes and so on) in his article [2] and the book [1]. 

Some examples of the periodic continued fractions were calculated in the papers [18], [20], 
and [21] by E. Korkina, [22] and [23] by G. Lachaud, [5], [30], [31], [32], and [33] by A. D. Bruno 
and V. I. Parusnukov, [11] and [12] by the author. A nice collection of two dimensional continued 
fractions was presented by K. Briggs, see [4]. 

0.1. Definition of periodic multidimensional continued fractions. In this section we 
recall some basic notions and definitions (see also [13]). Consider a space R n+1 (n > 1) over 
R. A point of R n+1 is said to be integer if all its coordinates are integers. Two sets are called 
integer- affine (integer- linearly) equivalent if there exists an affine (linear) transformation of R n+1 
preserving the set of all integer points, and transforming the first set to the second. A plane 
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is called integer if it is integer-affine equivalent to some plane passing through the origin and 
containing the sublattice of the integer lattice, and the rank of the sublattice is equivalent to 
the dimension of the plane. A polyhedron is said to be integer if all its vertices are integers. 

Consider an integer plane and an integer point in the complement to the plane. Let the 
Euclidean distance from the given point to the given plane equal The minimal value of 
nonzero Euclidean distances from integer points of the span of the the given plane and the given 
point to the plane is denoted by Iq. The ratio I/Iq is said to be the integer distance from the 
given integer point to the given integer plane. 

Consider arbitrary n + 1 hyperplanes in R n+1 that intersect at a unique point, namely the 
origin. The complement to the union of these hyperplanes consists of 2 ra+1 open orthants. Let 
us choose an arbitrary orthant. 

Definition 0.1. The boundary of the convex hull of all integer points except the origin in the 
closure of the orthant is called the sail of the orthant. The set of all 2 n+1 sails is called the 
n- dimensional continued fraction constructed according to the given n + 1 hyperplanes. 

Two n-dimensional continued fractions are said to be equivalent if the union of all sails of 
the first continued fraction is integer-linear equivalent to the union of all sails of the second 
continued fraction. 

Definition 0.2. An operator in the group SL(n + 1, Z) is called an integer irreducible hyperbolic 
operator if the following conditions hold: 

i) the characteristic polynomial of this operator is irreducible over Q; 

ii) all its eigenvalues are distinct and real. 

Consider some integer irreducible hyperbolic operator A € SL(n + 1,Z). Let us take the 
n-dimensional spaces that span all subsets of n linearly independent eigenvectors of the operator 
A. The spans of every n eigenvectors uniquely define n + 1 hyperplanes passing through the 
origin in general position. These hyperplanes uniquely define the multidimensional continued 
fraction associated to A. 

Definition 0.3. An n-dimensional continued fraction associated to some integer irreducible 
hyperbolic operator A is called an n-dimensional continued fraction of an (n + \)-algebraic 
irrationality. The case of n = 1(2) corresponds to one(two)- dimensional continued fractions of 
quadratic (cubic) irrationalities. 

Now we formulate the notion of periodic continued fraction associated an algebraic irrational- 
ity. Let A be an integer irreducible hyperbolic operator. Denote by H(j4) the set of all integer 
operators commuting with A. These operators form a ring with standard matrix addition and 
multiplication. (As a group S(A) is isomorphic to Z n+1 .) 

Consider the subset of the set SL(n + 1, Z) n E(A) that consists of all operators with positive 
real eigenvalues and denote it by 2(^4). From the Dirichlet unit element theorem (see. [3]) 
it follows that the subset S(A) forms a multiplicative Abelian group isomorphic to Z n , and 
that its action is free. Any operator of this group preserves the integer lattice and the union 
of all n + 1 hyperplanes, and hence it takes the n-dimensional continued fraction onto itself 
bijectively. (Whenever all eigenvalues are positive, the sails are also taken onto themselves in a 
one-to-one way.) In addition, the quotient of a sail under this group action is isomorphic to an n- 
dimensional torus. These statements are based on the generalization of the Lagrange theorem on 
ordinary continued fractions. The combinatorial topological generalization of Lagrange theorem 
was obtained by E. I. Korkina in [19] and its algebraic generalization by G. Lachaud [22]. 
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Theorem 0.4. (E. I. Korkina, [19].) Consider some orthant C and the sail K(C) corre- 
sponding to this orthant. Suppose that there exists a combinatorial isomorphism of the sail 
V(C) that preserves the combinatorial structure of the sail, then there exists an operator in the 
group GL(n + 1,Z) taking the orthant C and the sail V(C) to themselves and establishing the 
isomorphism. 

Unfortunately, the proof of this theorem is not yet published. 

The algebraic version of the generalized Lagrange theorem was obtained by G. Lachaud. The 
formulation of his theorem requires special notation and definitions, so we just refer the reader 
to the article [22], since we will not use it further. For more information on generalizations of 
the Lagrange theorem for the ordinary continued fractions to the multidimensional case see the 
papers [19], [38], [22] and [24]. 

By a fundamental domain of a sail we mean the union of some faces that contains exactly one 
face from each equivalence class (with respect to the action of the group S(A)). 

0.2. Different algorithms for constructing the sails of multidimensional continued 
fractions. A multidimensional periodic algebraic continued fraction is a set of infinite polyhe- 
dral surfaces (i.e., sails), that contain an infinite number of faces. As we have already mentioned, 
the quotient of any sail under the Dirichlet group action is isomorphic to an n-dimensional torus. 
The algebraic periodicity of the polyhedron allows to reconstruct the whole continued fraction 
knowing only the fundamental domain. Moreover, any fundamental domain contains only a 
finite number of faces of the whole algebraic periodic continued fraction. Hence we are faced 
with the problem of finding a good algorithm that enumerates all the faces for this domain. 

There were no algorithm for constructing multidimensional continued fractions until T. Shin- 
tani's work [34] in 1976. Let F be a totally real algebraic field of degree n. We take all different 
embeddings of F into R and denote them by <pi, i = 1, • • • , n (there are exactly n different 
embeddings, since F is totally real). Consider the following embedding of F into R n . For an 
arbitrary element x of F we suppose 

X -> ((p 1 (x),(p 2 (x), . . .,if n {x)). 

T. Shintani considered the action of the group of all totally positive elements for the ring of 
integers of F (by component-wise multiplication by totally positive integers x + ) on R™ for the 
described embedding of F. He proved that the fundamental domain for this action is the union 
of a finite number of simplicial cones of special type. (Note that if we take some other order for 
the embeddings (p?, then the fundamental domains will be integer- linear equivalent to the fun- 
damental domains for the embeddings considered above.) The statement of T. Shintani and its 
proof is actually the basis for the construction of one-dimensional continued fractions. Following 
T. Shintani's work, E. Thomas and A. T. Vasques obtained several fundamental domains for the 
two-dimensional case in [37] . Finally, R. Okazaki presented a method that permits to construct 
fundamental domains for fields of arbitrary degree in his article [29]. E. Korkina in [18], [20], [21] 
and G. Lachaud in [22], [23] produced an infinite number of fundamental domains for periodic 
algebraic two-dimensional continued fractions. The method used for constructing fundamental 
domains of multidimensional continued fractions in these papers was inductive. The method pro- 
duces the fundamental domain face by face, verifying that each new face does not lie in the same 
orbit with some face constructed before. Applying the method, one can find the fundamental 
domain in finitely many steps. 
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Later on J. O. Moussafir developed an essentially different approach in his work [28]. It works 
for an arbitrary (not necessary periodic) continued fraction and computes any bounded part 
of an infinite polyhedron. The approach is based on deduction. One produces a conjecture on 
the face structure for a big part of the continued fraction, then it remains to prove that any 
conjectured face is indeed a face of the part. This method can be also applied to the case of 
periodic continued fractions. 

In the present paper we describe a new advanced deductive construction adapted especially 
to fundamental domains of periodic continued fractions. The construction involves a method 
for conjecturing the structure of the fundamental domain and an algorithm testing whether the 
conjectured domain is indeed fundamental. The main advantage of our algorithm is the following: 
the number of "false" vertices of our approximation is much smaller than the number of "false" 
vertices of the approximation in the method of J. O. Moussafir (so that the computational time 
is considerable reduced). 

Note that this algorithm substantially uses the periodicity of the continued fraction and hence 
it is impossible to apply it to non-periodic continued fractions. 

We prove the following statement for the two-dimensional case. 

Suppose we have a conjecture on the structure of the fundamental domain for some sail of a 
two-dimensional periodic continued fraction. Let this domain contain N faces of all dimensions. 
The test of the conjecture (our algorithm) requires no more than CN A additions, multiplications 
and comparisons of two integers, where C is a universal constant that does not depend on N. 

All previous verification algorithms work exponential time with respect to N. 

Remark 0.5. Here we do not take into account that the integers can be quite large. To calculate 
upper bounds for the working time of the algorithms we need to multiply upper bounds for the 
number of operations by some polynomial of the coefficients of the matrix defining the continued 
fraction. 

Using the present algorithm, the author both generalized almost all known simple examples 
and series of examples of fundamental domains constructed before, and found a lot of new 
examples and series (see [11] and [12]). Using these examples, the author found the complete 
list of all two-dimensional periodic continued fractions constructed by matrices of small norm 
(|*| < 6) up to the integer-linear equivalence relation, see [13]. By the norm of a matrix, here 
we mean the sum of the absolute values of all its coefficients. 

0.3. Description of the paper. This work is organized as follows. The new method of sail 
construction consists of six steps. We discuss its plan in Section 1. In section 2 we describe the 
two common steps for both inductive and deductive methods. In this section we show how to 
find the generators for the group of SL(n, Z)-matrices commuting with the given one. All results 
of Section 2 are well-known and are given for completeness of exposition (see also the works [6] 
by Cohen and [23] by G. Lachaud). In Sections 3 and 4, we discuss the essential new part of 
the method. We show how to produce conjectures on fundamental domains in Section 3. In 
Section 4 we describe the algorithm for conjecture tests in the case of two-dimensional continued 
fractions. In that section, we also say a few words about the higher dimensional case. We 
conclude in Section 5 with the detailed study of one example of the method's application (see 
also [11]). 

Acknowledgement. The author is grateful to V. I. Arnold, E. I. Korkina, G. Lachaud, 
M. A. Tsfasman, and A. B. Sossinsky for constant attention to this work and useful remarks, 
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and to the Institut de Mathematiques de Luminy (CNRS) for hospitality and excellent working 
conditions. 

1. Description of the new construction 

1.1. Outline of the new construction. Now we briefly outline the main idea of the new 
construction of one of the fundamental domains of the multidimensional continued fraction 
corresponding to the given integer irreducible hyperbolic operator. Suppose that we are given 
the integer irreducible hyperbolic operator A £ SL(n + 1,Z). To compute some fundamental 
domain of a sail of the continued fraction associated to A it is sufficient to do the following: 

1. Compute a convex hull approximation of the sail. Namely take a large enough convenient 
set of integer points and find its convex hull. 

2. Make a conjecture on some fundamental domain. Here we need to guess a set of faces 
that might form a fundamental domain. We do this by finding a repeatable pattern in faces 
geometry. 

3. Prove the conjecture if possible. 

4. If cannot prove the conjecture, start with 1. but with a larger convenient set of points. 

In this situation the following two questions are actual: 
How to find a convenient set of integer points for the approximation of the sail? 
How to test whether the conjecture of a fundamental domain of the sail is true or not? 
We give the answers to these questions in the present paper. 

1.2. Steps of the construction. Let us briefly itemize the main steps of the method. 

The deductive algorithm of constructing one of fundamental domains for the sail 
of the given operator A in the given orthant. 

Step 1. Calculate the basis of the additive group of the ring E(A). 

Step 2. Calculate the basis of the group E(A) (using the result of Step 1). 

Step 3. Find some vertex of the sail. 

Step 4- Make a conjecture on a fundamental domain of the sail (using the results of Step 2 
and Step 3). 

Step 5. Test the produced (in Step 4) conjecture. 

Remark 1.1. It is supposed that the fundamental domain conjectured in Step 5 and the basis 
Ai, . . . , A n of the group £(A) satisfy the following conditions: 

i) the closure of the fundamental domain is homeomorphic to the disk; 

ii) the operators A\, . . . , A n define the gluing of this disk to the n-dimensional torus. 

Both inductive and deductive algorithms require the first and the second steps. We describe 
these two steps in the next section. All other steps are essential for our construction. In the 
method by J.-O. Moussafir [28] the conjecture has been producing using the approximation of 
the orthant by some rational orthant. In the present paper we propose to produce conjectures 
for some set of periods, see the description of Steps 3 and 4. We show how to test conjectures 
in the case of two-dimensional continued fractions in the description of Step 5. The result 
is partially based on the theorem on integer-affine classification of two-dimensional faces at 
the integer distances to the origin greater than one from [14]. (For the case of n-dimensional 
continued fractions for n > 3, the last step is quite complicated, since the classification of tree- 
dimensional faces at the integer distances to the origin greater than one is unknown.) In the 
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last step we also investigate an important particular case that seems to be quite common for 
periodic n-dimensional continued fractions as well. 

Remark 1.2. Note that all deductive algorithms are not algorithms in the strict sense. One 
should choose some basis of S(A) in the right way, produce a good conjecture, and then test 
it. Even the algorithmic recognition of the period for the given picture of the boundary of the 
sail approximation is supposed to be a hard problem. That is the reason why this "algorithm" 
cannot be done completely by some computer program. But at the other hand, the deductive 
algorithm is effective in practice. All of the examples listed in the article [11] were produced 
using this algorithm. The examples of this paper generalize and expand almost all known periods 
of the sails calculated before. 

Remark 1.3. The present method can be naturally generalized to the case of Minkowski- Voronoi 
model of multidimensional continued fractions. (For the definitions of Minkowski- Voronoi model 
see [26] and [39].) 

2. General questions concerning the lattice bases 

In this section we briefly discuss the questions which are necessary for both inductive and 
deductive methods (Steps 1 and 2 of Section 1). The answers to these questions were known 
before (see also [23] and [6]). 

2.1. Step 1. Calculation of a basis of the additive group of the ring E(A). 
Let VoVi . . . V m be some tetrahedron with vertices Vo, V\, . . . , V m -i, and V m . Denote by 
P(Vo, V\, . . . , V m ) the following parallelepiped: 

m 

[Vq + akWk\0 < a k < 1, k = 1, . . . m}. 

k=l 

In this section we consider 'B(A) as an additive group. We start the algorithm with the 
calculation of a basis for the group Let us identify the space Mat({n + 1) x (n + 1),M) 

with the space M (ra+1 ) and consider the standard metrics for this space. So any integer operator 
corresponds to some integer point, and the distance between two operators is the Euclidean 
distance between the corresponding points in R( ra+1 ) . We consider a sum of absolute values of 
all coefficients for some operator A as a norm of the operator A and denote it by \\A\\. 

In Proposition 2.2 below we show that the set E(A) is an additive group isomorphic to Z n+1 . 
Then by Corollary 2.3 below it follows that there exists a basis of the group 2(^4) contained in 
the parallelepiped P(0, E, A, A 2 , . . . , A a ). Thus by Proposition 2.4 below all norms of elements 
of the basis are bounded above by 

n 

N' = J2\\A% 

Remark 2.1. Applying the LLL-algorithm described by A. K. Lenstra, H. W. Lenstra and 
Jr., and L. Lovasz in [25] to the lattice generated by 0, E, A, A 2 , . . . , A n one constructs some 
reduced basis. This will decrease the number N' to some number N. (It is not necessary to use 
LLL-algorithm here, put just N = N'.) 

The set of integer operators contained in the parallelepiped is also a subset of the following 
set: 

Mat((n + 1) x (n + 1), Z) n B N (0), 
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where B^(0) — is an A^-neighborhood of the origin (i.e. the ball of radius N centered in the 
origin). The last set contains less no more than (2N + l) n+1 elements. This set contains all 
integer operators of the parallelepiped P(0, E,A,A 2 ,..., A n ). Further we find the set of integer 
points of the parallelepiped in a finite and polynomial with respect to iV number of operations 
(i.e. we have to choose a subset of all points of the parallelepiped in the set considered above). 
Finally we choose some basis of the group 2(^4) using the algorithm described in Proposition 2.5. 
Now we formulate the statements mentioned above. 

Proposition 2.2. For any integer irreducible hyperbolic operator A the set E(.A) forms an 
additive group isomorphic to Z n+1 . □ 

The detailed proof of Proposition 2.2 can be found, for instance, in the book [23] by 
G. Lachaud. 

Corollary 2.3. There exists a basis of the group S(A), such that all its elements are contained 
in the parallelepiped P(0, E, A, A 2 , . . . , A n ) . □ 

Proposition 2.4. Consider the parallelepiped P(0, E, A, A 2 , . . . , A n ). The norms of all opera- 
tors contained in this parallelepiped are bounded above by 

j=0 

The proof is straightforward. □ 

Proposition 2.5. Let there be given a maximal rank sublattice of the integer lattice in the integer 
plane. Let O; Ai, . . . , A n generate this sublattice (here O is the origin of the lattice). Then there 
exists a basis 0;Bi,..., B n of the integer lattice in the parallelepiped P(0, A±, . . . , A n ) such that 
for any natural i < n the vertex Bi belongs to the parallelepiped P(0, A±, . . . , Ai). 

Proof. We will inductively construct the basis O; B±, . . . , B n . On the i-th. step we will construct 
the basis O, B±, . . . , Bi inside the parallelepiped P(0, A±, . . . , Ai) that satisfies all the conditions 
listed above for the lattice in the plane spans the points O, A±, . . . , A4. 

Base of induction. For j = lwe choose the closest to the point O integer point as B\. 

Step of induction. Suppose we have constructed vertices O; B±, . . . , B^\ (i < n) satisfying the 
induction conditions. Now we construct an integer point -Bj. Let the integer distance between 
the point Ai and the plane spanned by the points 0,A±,... Ai_i be equal to Consider the 

plane 7Tj in the span of O, A\, . . . , Ai that is parallel to the plane spanned by O, Ai, . . . , A^\ and 
at the unit integer distance to that plane. The plane 7Tj is integer. Therefore the intersection of 
7Tj with the parallelepiped P(0, Ai, . . . , Ai) contains at list one integer point. We choose one of 
these points and denote it by B; L . 

As far as the spans of O, Ai, . . . , and O, B±, . . . , £?j_i coincides, all integer points of 
the parallelepiped P(0, B\, . . . , Bi) are at an integer distance or 1 to the plane spanning 
O, Bi, . . . , Bi-\. By induction assumption the points O; B\, . . . , generates the integer sub- 
lattice of the corresponding plane. Therefore the parallelepiped P(0, B\, ... , -B^-i) is empty (i.e. 
does not contain integer points different from vertices of the parallelepiped). From the last two 
facts it follows that the parallelepiped P(0, B±, . . . , Bi) is empty. Hence the points O; B±, . . . , Bi 
also generate the integer sublattice of the corresponding plane. 

So we have constructed the basis satisfying all the conditions of the proposition by induction. 

□ 
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The algorithm of this step does not seem to be the optimal one. So the following question is 
natural here: 

Problem 1. Find some effective algorithm of calculating an integer sublattice for the k- 
dimensional plane of the space W 11 if some basis A±, . . . , A^ for some sublattice of this plane 
is known (note that we know nothing about the corresponding quotient group). 

2.2. Step 2. Calculation of a basis of E(-A). From the algorithmic point of view this step 
is the most complicated. We describe only the idea for one of the simplest algorithms here and 
give the corresponding references. 

Let x( x ) De the characteristic polynomial of the operator A and let £ be one of the roots of 
x(x). Consider the following map 

/i:E(A)-Q[£]. 

For any element B £ S(j4) there exists a unique representation B = pb(A), where the degree of 
the polynomial ps is less than n+1 (since the operators E,A,...A n are linearly independent). 
We put 

h(B)=p B (0. 

Note that this map is an isomorphism between the ring E(A) and its image h(E(A)). The addi- 
tion and multiplication operations in the image are induced by the addition and multiplication 
operators of the field Q[£] (sec [23]). Moreover the set h(£(A)) forms an order in the field Q[£]. 
By the Dirichlet unit theorem it follows that there exist a basis for the units of this order and 
a number p, such that the norms of all its elements are bounded above by p. Since the method 
of constructing the constant p is standard, we omit it. (For the construction of p and proofs 
we refer to [3].) Note that the integer (n + l)-dimensional volume of the symplex spanned by 
the basis operators assign the minimal value. Now according to the book [3] we construct the 
basis by enumeration of all vectors of the set h(E(A)) inside the ball B p (0), where B p (0) is 
a p-neighborhood of the origin. The preimage (i.e. /i -1 ) of this basis gives us the basis of the 
group of invertible elements in the ring and hence it gives the basis of the subgroup 2(A). 

Remark 2.6. The constant p is extremely large (it equals the exponent of some polynomial 
of the coefficients of the matrix A). The effective algorithm for this step can be found in the 
book [6] written by H. Cohen. Using this algorithm one finds the basis of units in the polynomial 
(with respect to the coefficients of the matrix A) time. 

Remark 2.7. Note that it is not necessary to find generators of 'B(A). The algorithm works for 
arbitrary n + 1 linearly independent operators of 'B(A) (for more details see Remark 3.5 below). 

3. On fundamental domains and sail approximations 

The basis of the group E(A) was calculated in the previous section. Now we are coming to 
the main steps of the algorithm. In this section we calculate one of the sail vertices and show 
how to produce the conjectures. 

3.1. Step 3. How to calculate one of the sail vertices. First let us find some integer 
point of the orthant containing the sail. Consider an arbitrary orthant. Shift the standard unit 
parallelepiped inside this orthant. Some integer point lies inside the shifted parallelepiped. The 
coordinates of this point coincide with integer parts of coordinates for one of 2 n vertices of this 
parallelepiped. Using this fact one can easily find such point. 
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So we have found some integer point P of the orthant, let us find some vertex of the sail 
corresponding to this orthant. Consider some integer plane tt passing through the origin such 
that the intersection of tt with the orthant is at a unique point (at the origin). Suppose that 
an integer distance from the point P to this plane is equal to d. Now we look through all the 
symplexes obtained as intersections of our orthant with parallel to tt planes at integer distances 
to the origin equal 1, . . . , d. Suppose that the first symplex containing integer points lies in the 
plane at an integer distance equal d' < d. The convex hull of all points of this symplex coincides 
with some faces of the sail. All vertices of this face are vertices of the sail. Choose an arbitrary 
one of them. 

3.2. Step 4. How to produce a conjecture on the fundamental domain of a sail. 

Suppose that we know some point V of the sail in the orthant, and a basis A±, . . . , A n for the 
group E(j4). Now we must produce a conjecture on a fundamental domain of the sail. Let us 
briefly describe how to do this. First, we compute the set of integer points that contains all 
vertices of some fundamental domain of the sail. Secondly, we show how to choose the infinite 
sequence of special polyhedron approximation for the sail. Finally, using the picture of this 
approximations, we formulate a conjecture on a fundamental domain of the sail. 

Proposition 3.1. Let V be a vertex of the sail of n- dimensional continued fraction of a (n+ 1)- 
algebraic irrationality. Then there exists fundamental domain of the sail such that all vertices 
of this domain are contained in the convex hull H of the origin and of 2 n distinct points of the 
following form 



where Ei G {0, 1} for 1 <i <n. 



Proof. Consider the polyhedral cone C with vertex at the origin and base at the convex polyhe- 
dron with vertices V eit ...^ n . We take the union of all images of this polyhedral cone under the 
actions of operators 

n 

A — TT A m% 

^ 1 mi,...,m„ — 11 i ' 
i=l 

for 1 < i < n, where rrii G Z. Obviously this union is equivalent to the union of the whole open 
orthant and the origin. Therefore, any vertex of the sail is obtained from a vertex contained 
in the cone C by applying an operator A mi ^ for some integers m^. Moreover, the convex 
hull of all integer points of the given orthant contains the convex hull of the vertices of the form 
A mij ... tJnn (V). Hence the sail (i.e. the boundary of the convex hull of integer points) is contained 
in the closure of the complement in the orthant to the convex hull of all integer points of the 
form A rni) _^ mn {V). This complement is a subset of the union of polyhedra obtained from H 
by an action of some operator -A miv .. jrrin (for some integers mj, 1 < i < n). This concludes the 
proof of the proposition. □ 

We skip the classical description of the computation of the convex hull for the integer points 
contained in the polyhedron H. Denote the vertices of this convex hull by V r for < r < N 
(here N is the total number of such points). 

Definition 3.2. The convex hull of the following finite set of points 

{A mi> .... mn {V r )\l < nii < m,Mi : < i < n} 
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is called the n-th special polyhedron approximation for the sail. 

The defined set contains approximately and less than m n N points. (Since we calculated some 
image points for the boundary of H several times, we do not know the exact number of points.) 
The number N is fixed for the given generators A±, . . . , A n and m varies. We should try to make 
a good conjecture with the least possible m. 

Remark 3.3. For all the examples listed in the paper [11] (and for the example of the last 
section) it was sufficiently to take m = 2 to produce the corresponding conjectures. 

Remark 3.4. Note that N is a function defined on the set of all generators of the group H(^4) 
and it does not depend on m. Therefore the "quality" of the approximation also depends on the 
choice of the basis. 

Remark 3.5. Now we briefly discuss the case of Remark 2.7. Suppose we know operators 
Ai, . . . , A n that generate only some full rank subgroup of the group 'B(A). Let the index of this 
subgroup be equal to k. Then we are faced with the following two problems. First the number 
N will be approximately /c-times greater than in the previous case. Secondly one should also 
find a conjecture on generators of the group 

4. Test of the produced conjectures in the two-dimensional case 

Now it remains to test the produced conjectures of Step 4. In this section we explain how to 
test conjectures for the case of two-dimensional periodic continued fractions. The test consists 
of seven stages. It uses classification theorem from [14]. We prove here that these seven stages 
are sufficient for the verification whether the produced conjecture is true or not. The complexity 
of these stages is polynomial in the quantity all faces. 

4.1. Brief description of the test stages and formulation of the main results. Suppose 
we have a conjecture on some fundamental domain D for some sail of a two-dimensional periodic 
continued fraction associated to some integer irreducible hyperbolic operator A. Also from Step 2 
we know some basis B\,B2 of the group r.(A). Let pk (for k = 0, 1, 2) be the number of all k- 
dimensional faces of the fundamental domain D. Denote by Fi (i = 1, . . . ,P2) all two-dimensional 
faces, i. e. polygons. All adjacent to each face vertices and edges are known. It is also conjectured 
that the fundamental domain D and the basis #1,-62 satisfy the following conditions: 

i) the closure of the fundamental domain is homeomorphic to the two-dimensional disk; 

ii) B\ and B<i define the gluing of this disk to the n-dimensional torus (the fundamental domain 
D is in one-to-one correspondence with this torus). 

Test of the conjecture. Our test consists of the following seven stages: 

1. test of condition i); 

2. test of condition ii); 

3. calculation of all integer distances from the origin to the two-dimensional planes containing 
faces Fi and verification of their positivity; 

4. test on nonexistence of integer points inside the pyramids with vertices at the origin and 
bases at Fi (here the integer points of faces Fi are permitted); 

5. test on convexity of dihedral angles (for all edges of the fundamental domain); 

6. verification that all stars of the vertices are regular; 

7. test if all vertices of D are in the same orthant. 
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Theorem 4.1. The described conjecture test for the fundamental domain D requires less than 

C(p +Pi +P2) 4 

additions, multiplications and comparisons of two integers, where C is a universal constant that 
does not depend on p^ . 

Remark 4.2. Note that here we do not take into account the complexity of additions, multipli- 
cations and comparison of two large integers. We think of any such operation as of one operation 
(as a unit of time). There are known some linear with respect to the coefficients of the matrix 
A bounds for the number of digits of such integers. So the complexity should be multiplied by 
some polynomial of the coefficients of A. 

Theorem 4.3. Let the set of faces D satisfy the following conditions: 

1) condition i); 

2) condition ii); 

3) positivity of all integer distances from the origin to the two-dimensional planes containing 
faces Fi; 

4) there are no integer points inside the pyramids with vertices at the origin and bases at Fi 
(here the integer points of faces Fi are permitted); 

5) all dihedral angles are convex; 

6) all stars of the vertices are regular; 

7) all vertices of D are contained in the same orthant. 

Then D is a fundamental domain of some sail of the continued fraction associated to the opera- 
tor A. 

We start with the proof of Theorem 4.1. Let us show that all the stages listed above can be 
realized in the polynomial time. 

4.2. Test of condition i). First we need to test that the closures of any two two-dimensional 
faces either do not intersect or intersect at a vertex, or at an edge (and hence the closures of any 
two one-dimensional faces either do not intersect or intersect at a vertex) . For this test we need to 
solve linear systems of two equations and 2p\ inequalities that define faces Fi (in three variables) . 
The number of such systems equals the number of couples of faces, i.e. equals P2 ^ p | l \ Since 
there are only three variables and two linear equalities, any system can be reduced to a system 
of inequalities in one variable in a linear of p\ time. This implies that we need no more than 
C\^\p\p\ operations to solve all systems, where C\^\ is some constant that does not depend on 

Pi- 

Secondly we have to test that the edges are adjacent either to one or to two (two-dimensional) 
faces. This can be done in less than P1P2 single adjacency tests. Any adjacent test can be done 
in a linear of p\ time. This yields that we need no more than C\^p\p2 operations for these tests, 
where is some constant that does not depend on p, L . 

Consider all (one-dimensional) edges of the closure of D that are adjacent to exactly one face 
of D. Thirdly we test that the union of such edges is homeomorphic to the circle, i. e. is piece- 
wise connected and does not have self intersections. We will check that this set is piece-wise 
connected, and that for any vertex of the closure of D either exactly two of the described edges 
are adjacent to the vertex or none of the described edges is adjacent to it. The test contains 
no more than p\ simple adjacency tests. Any simple adjacency test requires a finite number of 
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operations that does not depend on pi for i = 0, 1, 2. So, we need no more than C\$p\ operations 
for these tests, where 61,3 is some constant that does not depend on pi. 

Further we will check that the union of the closures of all faces of D is piecewise connected. 
Let us find explicitly one of the connected components. Consider an arbitrary face and all 
its edges and vertices. Take all faces whose closures contain these edges and vertices (except 
the first face) and consider all new edges and vertices at their boundaries. We continue our 
construction inductively. Suppose we have made I steps and constructed some part of the 
connected component. Consider again all new (appeared on the i-th step) edges and vertices on 
its boundary. Take all faces whose closures contain these edges and vertices (except old faces). 
Since any edge of D considers no more than once, this algorithm requires no more than 6{ 4 pi 
simple adjacency tests. So we know one of the connected components. Now it remains to check 
that all faces are in this component and all edges and vertices are adjacent to these faces. Finally 
we need no more than C\^{po +p\ + P2) operations. 

Suppose all tests of this subsection are positive. So the closure of D is piecewise connected, 
and its boundary is homeomorphic to the circle. The closure of D is homeomorphic to the 
two-dimensional disk iff its Euler characteristics equals one. We need less than 4po + 2pi +P2 of 
additions for the calculation of Euler characteristics for the closure of D. 

We have proved the following lemma. 

Lemma 4.4. The first stage of the test requires no more than 

Ci{plp2 + Pipl + Po) 

additions, multiplications and comparisons of two integers, where C\ is a universal constant that 
does not depend onpi. □ 

4.3. Test of condition ii). First we check that after the gluing of the disk D we obtain a 
nonsingular triangulated surface. 

Let us test that there is no singular points inside the disk, i. e. open faces of the interior of 
the disk and their images after gluing do not intersect. This requires 2(po +p\ + P2) 2 systems of 
linear equations and inequalities. Any such system can be solved in a linear time with respect to 
Po, pi, pi- Hence we need no more than C2,i(po +Pi +P2) 3 operations, where 6*2,1 is a universal 
constant that does not depend on p^. 

Now we test that there is no singularities on edges on the boundary of the disk. Here we 
need to check that for any edge of the boundary there exists a unique edge that is gluing with 
the first edge. This requires no more than p\ adjacency tests in 62,2 operations each. Hence we 
need no more than C^P? operations (6*2,1 does not depend on pi). 

Further we check that there is no singularities at the vertices (after the gluing). For any vertex 
we need to check that the union of all faces and edges containing the vertex in their closure is 
homeomorphic to the disk. Now we know that faces of the fundamental domain do not intersect. 
It remains only to test that any edge is adjacent to exactly two faces (after gluing), and that the 
union of all faces and edges containing the vertex in their closure is piece wise connected and 
orientable. The test of all these conditions requires no more than hp\p2 adjacency tests for some 
constant h. Thus to test the vertex we need no more than C2flP\p\ operations (6*2,3 does not 
depend on pi). For all vertices of the boundary we need no more than 62,3^0^1^2 operations. 

Finally, let us check that this surface is homeomorphic to the torus. First the Euler character- 
istic of the surface should be zero (to verify this we need a linear of pi time) . Secondly we show 
the orientability of the surface. We orient the boundary circle and check that any two boundary 
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edge that we glue together are glued with the opposite orientations. If for one of the couples of 
such edges the orientation coincides, we have a Klein bottle. Otherwise, we get a torus. This 
requires a linear of p\ number of operations. Therefore for this test we need less than or equal 
C2,a{pq + Pi + P2) operations (6*2,4 does not depend on pi). 

It remains to show that the fundamental domain D maps to the obtained torus bijectively. 
For all faces this holds automatically. The corresponding test for edges and vertices requires 
62,5 (po + Pi) operations (6*2,5 does not depend on pi). 

We have proven the following lemma. 

Lemma 4.5. The second stage of the test requires no more than 

C 2 (poPiP2 + (Po + Pi + P2) 3 ) 

additions, multiplications and comparisons of two integers, where C2 is a universal constant that 
does not depend on p, t . □ 

4.4. Calculation of all integer distances from the origin to the two-dimensional planes 
containing faces Fj, their positivity. Let V\{x\, y±, z\), ^2(^2,2/2,^2), and ^3(^3, y 3 , z 3 ) be 
some integer points that do not lie in a straight line. Then the following statement holds. 

Proposition 4.6. An integer distance from the origin to the two-dimensional integer plane that 
spans the points V±, V2, and V3 is equal to 

xi x 2 x 3 \ 

yi V2 2/3 

Zl Z2 Z 3 J 

\[V2^V1, V3- VI] | z ' 

where by [V2 — VI, V3 — VI] we denote the cross product of the vectors V2 — VI and V3 — VI in 
R 3 , and by \[V2 - VI, V3 - Vl]|z we denote the integer lengths of the vector [VT^V1,V3^VI] 
(i. e. the greater common divisor of the coordinates of this vector). □ 

The proof is straightforward and is omitted here. 

Remark 4.7. This proposition can be generalized to the higher dimensional case. 

Lemma 4.8. The third stage of the test requires no more than C3P2 additions, multiplications 
and comparisons of two integers, where C3 is a universal constant that does not depend onpi. □ 

Proof. Any integer distance to the origin can be found by the formula of Proposition 4.6 (this 
requires some finite number of steps that does not depend on pi). □ 

4.5. Test on nonexistence of integer points inside the pyramids with vertices at the 
origin and bases at Fj. First we formulate the following integer-linear classification theorem. 

Theorem 4.9. (See [14].) Any compact two-dimensional face of a sails of a two-dimensional 
continued fraction contained in a plane at an integer distance to the origin greater than one is 
integer-linear equivalent exactly to one of the faces with vertices of the following list: 

- (£, r — 1, —r), (a + £, r — 1, — r), (£, r, — r), where a > 1, and £ and r are relatively prime, 
and r > 2 and < £ < r/2; 

- (2, 1,6- 1), (2, 2, -1), (2, 0, -1), where b > 2; 

- (2,-2,1), (2,-1,-1), (2,1,2) and (3,0,2), (3,1,1), (3,2,3). □ 
For this stage we will also need the following proposition. 
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Proposition 4.10. Consider two integer triangles in M 3 with vertices A±(x ai , y ai , z ai ), 
M{x a2 ,y a2 ,z a2 ), A 3 (x as , y as , z as ) and with vertices Bi(x bl ,y bl ,z bl ), B 2 (x b2 , y b2 , z h2 ) , 
B3 (x b3 , y b3 , z b3 ) . Let the plane of the first triangle do not contain the origin. Then the 
triangle A1A2A3 is integer- linear equivalent to the triangle B1B2B3 (in the corresponding order) 
iff the absolute value of the determinant of the matrix BA -1 equals one, and all the coefficients 
of this matrix are integers, where 



We left the proof of this proposition as an easy exercise to the reader. 

Lemma 4.11. The fourth stage of the test requires no more than C4P2 additions, multiplications 
and comparisons of two integers, where C4 is a universal constant that does not depend on pi . 

Proof. Consider some face Fi. If an integer distance from the origin to the plane containing 
the face equals one then all integer points of the pyramid except the vertex at the origin are 
contained in the base. 

Suppose now that an integer distance from the origin to the plane containing the face equals 
rj > 1. By Theorem 4.9 it follows that this pyramid is triangular. First we calculate integer 
lengths of the edges and the integer area of Fi (in a fixed number of operations). Further by 
Proposition 4.10 the verification of an integer- linear type of the face is reduced to the solution 
of nine integer linear equations on £ or to the verification of some nine rationals to be integers. 

Therefore for any face we need some constant number of operations C4 (where C4 does not 
depend on pi). Thus the complexity of the fourth stage of the test is no more than C4p 2 - This 
concludes the proof of the lemma. □ 

4.6. Test of the convexity of dihedral angles. A dihedral angle is called well-placed if the 
origin is contained in the corresponding opposite angle. 

Note that the property of some angle to be well-placed is an integer- linear invariant. Thus it 
is sufficient to check this property for all dihedral angles for edges of the closure of D. The test 
for each edge reduces to the solution of a system of ordinary inequalities (without variables). 

Lemma 4.12. The fifth stage of the test requires no more than C^p\ additions, multiplications 
and comparisons of two integers, where C5 is a universal constant that does not depend onpi. □ 

4.7. Verification that all 2-stars of the vertices are regular. Let p : W — > T 2 be the 

universal covering of the torus after the gluing. By the 2-star at the vertex of the universal 
covering of the torus we will call the union of all faces of the universal covering (of dimensions 
no more than 2) to which the given vertex is adjacent. 

Let a vertex v € W map to x G R 3 . The faces of the universal covering to which the given 
vertex is adjacent maps to the faces with the same property. 

If x / (a, 0,0) for some positive a, then by v n we denote the vector (1/n, 0,0), n <G N. (If 
x = (a, 0, 0), then by v n we denote (0, 1/n, 0).) A 2-star at v is called regular if for the sequence 
of rays l n passing through the point x + v n and having vertexes at the origin there exists a 
positive k such that for any m > k the following holds: the preimage (p _1 ) of the intersection 
of the image of 2-star at v and l m in the universal covering consists of exactly one point. 
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Lemma 4.13. The sixth stage of the test requires no more than 

CePoPiiPo + P2) 

additions, multiplications and comparisons of two integers, where Cq is a universal constant that 
does not depend on pi . 

Proof. For any face (edge) of the image of the 2-star for any vertex we need to solve a system 
of no more than p\ (po, respectively) linear inequalities of the variable e = 1/n. Hence the 
complexity of the test is no more than to 

CePoiPoPi + P1P2), 

where Cq is a universal constant that does not depend on pi. □ 

4.8. Test for all vertices of D to be in the same orthant; conclusion of the proof of 
Theorem 4.1. 

Lemma 4.14. The seventh stage of the test requires no more than C7P0 additions, multiplica- 
tions and comparisons of two integers, where C-j is a universal constant that does not depend 
on pi. 

Proof. We will use the following statement. Let e be some nonzero vector. The volume of the 
parallelepiped generated by the vectors e, A(e), and A 2 (e) equals zero iff the vectors e, A(e), 
and A 2 (e) generate some eigensubspace of A (of nonzero codimension). 

Consider two vertices x\ and X2 of D. Let x(t) = tx\ + (1 — t)x2- The vertices x\ and X2 
are contained in the same orthant iff the volume function f(t) of the parallelepiped generated 
by the vertices x(t), Bi(x(t)), and Bf(x(t)) does not have zeros in the segment [0, 1]. Note that 
f(x) equals the determinant of the matrix generated by the vectors x(t), Bi(x(t)), and B\{x{t)). 
Thus f{x) is a polynomial of the third degree with integer coefficients. All solutions of f(x) = 
can be found explicitly. It remains to compare them with and 1. 

Let us fix x\ and vary X2 (in the set of all vertices of D). The corresponding test requires no 
more than C-jpo operations (C7 does not depend on pi). □ 

Remark 4.15. As long as the integer operator B\ is hyperbolic and irreducible and all the 
coefficients of f(x) are integers, it follows that f(x) has three distinct real roots. So the question 
of existence of roots in [0, 1] can be reduced to calculating critical points, comparing them with 
and 1, and comparing the critical values with /(0) and /(l). Here we solve only one quadratic 
equation (instead of the cubic one f{x) = 0). 

Conclusion of the proof of Theorem 4-1- From Lemmas 4.4 — 4.14 it follows that all seven stages 
of the test require no more than C(po + Pi + P2) 4 additions, multiplications and comparisons 
of two integers, where C is a universal constant that does not depend on pi. The proof of 
Theorem 4.1 is complete. □ 

Remark 4.16. Actually a stronger statement holds. The whole test requires no more than 
C(po +p\ +P2) 3 additions, multiplications and comparisons of two integers, and C(po +Pi +P2) 4 
logical operations, where C and C are universal constants that do not depend on p^. 

It remains to prove that this seven stages are sufficient for the test. 
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4.9. Lemma on the injectivity of the face projection. We prove Theorem 4.3 in four 
lemmas. 

First let us give the necessary notation. Let the operators B\ and B<i generate For any 

integers n, m by B n ^ m we denote an operator B^B™. We suppose that our domain D satisfies 
conditions 1 — 7 of Theorem 4.3. Let 

U= [J B n>m {D). 

n,mgZ 

Consider the unit two-dimensional sphere S 2 centered at the origin O. We denote by ir the 
following map: 

vr : M 3 \ O -» S, 

where any point x G M 3 \ O maps to the point at the intersection of S 2 and the ray with vertex 
at the origin and containing x. 

Lemma 4.17. For any face of the polygonal surface U the map tt is well-defined and injective 
on it. 

Proof. Consider any two-dimensional face F of the surface U. By condition 3, the distance from 
the origin to the plane containing F is greater than zero. Hence this plane does not contain the 
origin. Then tt is well-defined and injective on F. 

Let now E be some edge of U. By conditions 1 and 2, this edge is adjacent to some two- 
dimensional face and therefore is contained in some plane that does not pass through the origin. 
Thus the line containing E does not pass through the origin. Hence tt is well defined and injective 
on E. 

The injectivity for the vertices is obvious. □ 

4.10. Lemma on the finite covering of the fundamental domain. Let 1 £ K 3 \0. Denote 
by N x the tetrahedral angle with vertex at the origin and base with vertices x, B\(x), B\B2(x), 
and B2(x). Notice that 

U B n>m (N x ) \ O 

\n,m£Z J 

is one of eight orthants of the continued fraction associated to A that contains x. Note that 
from conditions 1, 2, and 6 it follows that all points of D are contained in one open orthant, we 
denote it by K. 

Lemma 4.18. Let x be some point of the open orthant K. Then the union of all faces of D is 
contained in a finite union of solid angles of the type B n>m (N x ). 

Proof. By the Dirichlet unit theorem [3] it follows that for any interior point a of the open orthant 
K there exists an open neighborhood satisfying the following condition. The neighborhood can 
be covered by four solid angles of the type B n ^ m {N x ) when a belongs to an edge of some Bkj(N x ); 
by two solid angles when a belongs to the face of some Bkj(N x ); and by one solid angle in the 
remaining cases. In any case the neighborhood can be covered by some finite union of solid 
angles of the type B n)m (N x ). 

Consider a covering of D by such neighborhoods that correspond to each point of the closure 
of D. Since the closure of D is closed and bounded in M 3 , it is compact. Hence this covering 
contains some finite subcovering. Therefore the union of all faces of D is contained in the finite 
union of solid angles of type B n ^ m (N x ). The proof is complete. □ 
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Corollary 4.19. Let x be contained in the open orthant K. Then the solid angle N x contains 
only points from a finite number of fundamental domains of the type B ntTn (D). 

i 

Proof. From the last lemma it follows that D is contained in the finite union (J B nkjTnk (N x ) (for 

k=\ 

some positive I). Then the solid angle N x can contain only points of the fundamental domains 
B_ nfc _ rofc (D) for 1 < k < I. □ 

4.11. Lemma on the bijectivity of the projection. 

Lemma 4.20. The map tt bijectively takes the polygonal surface U to the set S 2 n K . 

Proof. As it was shown above, the surface U is contained in K and is taken to S 2 n K under the 
map tt. 

Let us introduce the following notation. By condition 2, the operators B\ and B2 glue the 
fundamental domain into the torus T 2 . Let W be the universal covering of T 2 . The face 
decomposition on T 2 lifts to a face decomposition on W. There is a natural two-parametric 
family (with two integer parameters) of projections p n ,m '■ W — > U that maps faces to faces 
(since the group of shifts Bk t i acts on U). Let us choose one of these projections and denote it 
by p (p : W — > U) . 

Consider the map tt o p : W — > S 2 . This map does not have branch points at the images of 
open faces of W, since any face of W bijectively maps to some face of U, and the corresponding 
face of U injectively maps to S 2 n K by Lemma 4.17. 

Two faces with common edge of W map to some two faces with common edge of U, such faces 
of U generate a well-placed dihedral angle, and hence also injectively map to S 2 n K. So the 
map tt o p does not have branch points at the images of open edges. 

Now we consider some vertex v oi W . The edges and faces of W with common vertex v 
by condition 6 form a regular 2-star. This edges also maps to some edges of U with common 
vertex. Thus there exist a sequence of points that tends to tt op(y) (contained in S 2 ) such that 
the preimage of any point of the sequence has exactly one preimage in the 2-star of v. Hence 
tt op does not have branch points at the sheet containing the star at tt op(v). Therefore, nop 
does not have any branch points at the vertices. 

So the map nop: W — > S 2 (1 K does not have branch points. 

Consider an arbitrary point x G S 2 H K and the solid angle N x corresponding to it. Let 
x\ and X2 be two points of S 2 n N x . Now we will show that the preimages (tt opj^f^) and 
{~nop)~ l (x2) contain the same number of points. Let us join the points x\ and X2 by some curve 
inside S 2 fl N x . By Corollary 4.19, we know that the preimage of this curve is contained in a 
finite number of faces of W . Since there are no branch points in any face (and their number is 
finite) and there are no boundary faces of W, the number of preimages for tt op is some (finite) 
discrete and continuous function on this curve. Therefore the number of preimages for it o p of 
any two points of S 2 n N x is the same. Hence the number of preimages for tt o p of any two 
points of S 2 fl K is the same. 

From this we conclude that the projection tt o p of the universal covering W (homeomorphic 
to an open disk) to S 2 fl K (i. e. homeomorphic to an open disk) is a nonramified covering 
with finitely many sheets. Since the covering (of an open disk by an open disk) is piece-wise 
connected, the number of sheets equals one. Since by definition p : W — > U is surjective and 
by the all above it is injective, the maps p : W — > U and tt : U — > S 2 f~l K are bijective. This 
concludes the proof of the lemma. □ 
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4.12. Lemma on convexity. Since any ball centered at the origin contains only a finite number 
of vertices of U (and they do not form a sequence tending to the boundary of the orthant K), 
the polyhedral surface U divide the space M 3 into two connected components. Denote by H the 
connected component of the complement to U that does not contain the origin. 

Lemma 4.21. The set H is convex. 

Proof. Suppose that some plane passing through the origin intersects the polygonal surface U 
and does not contain any vertex of U. By Lemma 4.20 such plane intersects U at some piecewise 
connected broken line with an infinite number of edges. The complement of the plane to this 
broken line consists of two connected components. By assumption all vertices of this broken line 
are contained in open edges of U. By condition 5 all dihedral angles of U are well-placed. Thus 
the angle at any vertex of intersection of H with our plane is less than straight angle. Hence by 
the previous lemma the intersection is convex. 

Consider the set of all planes that pass through the origin, intersect U and do not contain 
vertices of U. This set is dense in the set of all planes passing through the origin and intersecting 
U. Therefore by the continuity reasons it follows that the intersection of H with any plane 
passing through the origin (and intersecting U) is convex. 

Now we prove that the set H is convex. Let x\ and X2 be some points of H. Consider the 
plane that spans xi, X2, and the origin. This plane intersects U since x\ is in H and the origin 
is not in H. By the above the intersection of H with this plane is convex. Hence the segment 
with endpoints X\ and x<i is contained in H. Thus H is convex (by the definition of convexity). 

Lemma 4.21 is proven. □ 

4.13. Conclusion of the proof of Theorem 4.3: the main part. So the constructed polyg- 
onal surface U possess the following properties: 

— by Lemma 4.21 U bounds the convex set H; 

— by construction all vertices of U are integer points; 

— by condition 4 the set K \ H does not contain integer points. 

Therefore the polygonal surface U is the boundary of the convex hull of all integer points 
inside K. Thus by definition U is one of the sails of the continued fraction associated to the 
operator A. This concludes the proof of Theorem 4.3. □ 

Let us formulate one important conjecture here. 

Conjecture 2. Conditions 1 — 6 imply condition 7. 

4.14. On the verification of the conjecture for the multidimensional case. Here we 
briefly outline an idea how to test the conjecture for fundamental domains of multidimensional 
continued fractions. 

Conjecture for the multidimensional case. Suppose we have a conjecture on some fundamental 
domain D, and also some basis Bi,...,B n of the group S(A). Let also the fundamental domain 
and the basis posses the following properties: 

i) the closure of the fundamental domain is homeomorphic to the disk; 

ii) the operators B\, . . . B n define the gluing of this disk to the n-dimensional torus. 

How to test the conjecture for fundamental domains of multidimensional continued fractions ? 
The verification of conditions i) and ii) is straightforward and is omitted here. If these conditions 
hold we verify if all the n-dimensional faces of the fundamental domain are faces of the sail. It 
can be done in the following way. 
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Suppose that integer distances from the origin to the planes of faces F{ are equal to di (i = 
1, . . . ,p, where p — is the number of all n-dimensional faces). Our conjecture is true iff for all 
i = 1, . . . ,p the following conditions hold. 

a) For any integer d < di consider the plane parallel to the face Fi with integer distances to the 
origin equals d. The intersection of our orthant with this plane does not contain any integer 
point. 

b) For d = di the convex hull of all integer points in the intersection coincides with face Fi. 
The verification of conditions a) and b) is quite complicated from the algorithmical point of 
view. 

We conclude this section with the important inverse question of construction periodic contin- 
ued fractions. 

Problem 3. (V. I. Arnold.) Does there exist an algorithm to decide whether a given type of 
fundamental domain is realizable by a periodic continued fraction. 

The answer to this question is unknown even for the two-dimensional periodic continued 
fractions. 

5. An example of the calculation of a fundamental domain 

The example of two-dimensional sails in this section was announced in the article [11] by the 
author. For arbitrary integer numbers m and n we denote by A m ^ n the Sylvester operator 






1 











1 


1 


— m 


— n 



We construct fundamental domains for some particular two-dimensional subfamily of the two- 
dimensional continued fractions corresponding to Sylvester operators for the orthant that con- 
tains the point (0,0, 1). 

Theorem 5.1. Let m = b — a — 1, n=(a + 2)(6 + 1) (where a,b > 0). Consider the sail 
of the operator A m ^ n containing the point (0,0,1). Let A = (1,0, a + 2), B = (0,0,1), C = 
(b — a — 1, 1, 0), and D = ((b + l) 2 , b + 1,1). Then the following set of faces forms one of the 
fundamental domains: 

1 ) the vertex A; 

2) the edges AB, AD, and BD; 

3) the triangular faces ABD and BDC. 

The closure of the fundamental domain is homeomorphic to the square shown on Fig. 1 (for 
the case of an arbitrary a, and 6 = 6). 

Proof. Steps 1 and 2. We omit the first and the second steps (these steps are classical, see [6])) 
and here write down the result. The following two operators generate the group S(A): 

X a ,b = A mn , Y a ^ = A mn (A mr j — (6 + 1)/), 

where I is the identity element in the group SL(3, Z). 

Step 3. We prove that (0, 0, 1) is a vertex of the sail. Consider the plane passing through A, 
B, and D: 

(-1 - a)x + (ab + a + b + l)y + z = 1. 
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a,b>0 



Figure 1 . The closure of the fundamental domain of a sail of a fraction associ- 
ated to the operator ^4b-a-i,(aH-2)(6+i) (here 6 = 6, and a is arbitrary). 



As far as the equations (in variables x, y, and z) 



(— 1 — a)x + (ab + a + 6 + l)y + z = a 



do not have any integer solution for < a < 1, an integer distance from ABD to the origin is 
equal to one. There are exactly three integer points (A, B, and D) in the intersection of the 
plane and the orthant (we left the proof of that fact for the reader as an exercise) . 

Step 4- The conjecture of the fundamental domain was produced in the statement of this 
theorem. 

So it remains to complete Step 5: to test the conjectured fundamental domain. For the test 
we need some extra points: 



1. (Test of condition i).) It can be shown in the usual way that the faces have the common 
edge BD, and the edges intersect only at vertices. This implies that all adjacencies are correct, 
and that only one or two faces are adjacent to each edge. The closure of the boundary is a closed 
broken line ABCDA, homeomorphic to the circle. 

2. ( Test of condition ii).) By direct calculations it follows that by the operator X a ^ action the 
segment AB is taken to the segment DC (the point A maps to the point D and B to C) and by 
the operator Y a ^ action the segment AD is taken to the segment BC (the point A maps to the 
point B and D to C). Obviously that no other points glue together. The Euler characteristic of 
the obtained surface equals 2 — 3 + 1, i.e. zero, and the surface is orientable. 

3. (Calculation of all integer distances from the origin to the two-dimensional planes con- 
taining faces.) Let us calculate integer distances from the origin to the two-dimensional planes 
containing faces ABD and BDC by the formula of Lemma 4.6. An integer distance to the plane 
of ABD equals 



E 



X~l(B) = (1, -ab-a-2b- 2, 



a 2 b 2 + 2a 2 b + 4ab 2 + a 2 + 8ab + Ab 2 + 5a + 7b + 5); 
r Oi6 (B) = (-a-2,l,0); 

X~l(F) = (0, -b - 1, ab 2 + lab + 2b 2 + a + 46 + 3). 



F 
H 
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An integer distance to the plane of BDC equals 

b 2 + 2b+l b-a-1 

6 + 1 1 

1 1 



6+1 



ab + 26 + a + 2 

6 + 1 = a + 2 - 





^. {Test on nonexistence of integer points inside the pyramids with vertices at the origin and 
bases at the faces.) Since the integer distance from the origin to the plane containing ABD 
equals one, the pyramid corresponding to ABD does not contain integer points different from 
O and the points of the face ABD. 

The face BDC is integer-linear equivalent to the face with vertices (l,a + 1, —a — 2), (6 + 
2, a + 1, —a — 2), (1, a + 2, —a — 2) of the list of Theorem 4.9. The corresponding transformation 
taking BCD to the face of the list of Theorem 4.9 is the following: 

b-a-1 
1 

-1 

By Theorem 4.9 the pyramid corresponding to BDC does not contain integer points different 
from O and the points of the face BDC. 

5. (Test on convexity of dihedral angles.) Let us first consider the edge BD. This edge is 
adjacent to the faces ABD and BDC. The face ABD is contained in the plane fABD(x, y, z) = 0, 
and the face BDC is contained in the plane fsDc( x , U, z) = 0, where 

fABD{x,y,z) = (-1 - a)x + (ab + a + 6 + l)y + z - 1; 
f B Dc(x,y,z) = x + (b + l)y - (a + 2)z + (a + 2). 

To test that the dihedral angle corresponding to the edge BD is well-placed it is sufficient to 
verify the following: the point C and the origin O lie in different half-spaces with respect to 
the plane that spans the points A, B, and D; the points A and O lie in different half-spaces 
with respect to the plane that spans the points C, B, and D. So we need to solve the following 
system: 

' fABD(C)- f ABD (0) < 
fBDc(A) ■ fBDc(O) < • 

This system is equivalent to the following one: 

(a 2 + 3a + 2) • (-1) < 
(-a 2 -3a- 1) • (a + 2) < ' 

Since a > 0, the inequalities hold. Thus the dihedral angle associated with the edge BD is 
well-placed. 

Since the cases of dihedral angles associated to the edges AB (and the faces ADB and AEB) 
and BC (and the faces BDC and CBF) can be verified in the same way, we omit their descrip- 
tions. 

This concludes the test of condition 5. 

6. (Verification that all 2- stars of the vertices are regular.) There is only one vertex in the 
torus decomposition. Any lift of this point to the universal covering W is adjacent to six edges 
and six faces. Consider a vertex of the universal covering that maps to the point B. The 
corresponding 2-star maps to six edges BC, BD, BA, BE, BH, and BF and to six faces BCD, 
BDA, BAE, BEH, BHF, and BFC, where 

H = X~\(F) = (0, -6 - 1, ab 2 + 2a6 + 26 2 + a + 46 + 3). 
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We will check that for any sufficiently small positive e a ray l £ with vertex at the origin and 
passing through the point P £ = (e, 0, 1) intersects the exactly one of the faces of 2-stars. 

First we will check that for any sufficiently small positive e the ray l £ intersects the triangle 
BCF. Or, equivalently, that the ray l £ is contained in the trihedral angle with vertex at the origin 
O and base in the triangle BCF. The two-dimensional face of the trihedral angle containing 
B, C, and O can be defined by Jabo = 0; the two-dimensional face of the trihedral angle 
containing B, F, and O can be defined by Jbfo = 0; the two-dimensional face of the trihedral 
angle containing C, F, and O can be defined by fcFO = 0, where 

fBCo{x,y,z) = x+(a + l-b)y; 
fBFo{x,y,z) = x+(a + 2)y; 
fcFo{x,y,z) = z. 

For any sufficiently small positive e the ray l £ is contained in the dihedral angle defined above if 
the following conditions hold: the points P £ and F are in the same closed half-space with respect 
to the plane Jbco = 0; the points P £ and C are in the same closed half-space with respect to the 
plane /bfo = 0; the points P £ and B are in the same closed half-space with respect to the plane 
fcFO = 0. Since the points P £ and B are close to each other for sufficiently small e, they are 
in the same closed half-space with respect to the plane fcFO = 0. Now we check the remaining 
two conditions: 

/ fscoiPe) ■ fBCo(F) > / (-b-l)e > 

1 fBF0(Pe) ■ fBFoiC) > \ (b + \)e > " 

Since b, e > the first inequality does not hold. Thus for any sufficiently small positive e the 
ray l £ does not intersect the triangle BCF. 

The cases of the triangles BCD, BDA, BAE, BEH, and BHF are similar to those described 
above and are omitted here. 

The ray l £ (for any sufficiently small positive e) intersects the bijective image of a 2-star of 
the vertex at exactly one point contained in the edge AB. Therefore all 2-stars associated to 
the vertices are regular. 

7. (Test that all the vertices of D are in the same orthant.) The test of the seventh stage for 
this theorem is trivial since D contains exactly one vertex. 

This concludes the proof of Theorem 5.1. □ 
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